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Introduction 

This paper will discuss the development of evaluation strategies for a non-profit 
child abuse prevention agency in Maryland. The Family Tree, an organization associated 
with National Parents Anonymous and the National Exchange Club Foundation for the 
Prevention of Child Abuse, serves over 15,000 individuals per year through community 
training, parenting education, and home visitation programs. With the participation of its 
staff. The Family Tree identified an overall agency outcome plan. In addition, each of the 
agency’s four core programs created linking outcome logic models. A participatory 
approach was used throughout the process with facilitation from an outside consultant. 
What has resulted is a new way of doing business and a fresh perspective on program 
planning. 

Literature Review 

The Problem of Child Maltreatment 

Child maltreatment is an issue of national importance that has a detrimental 
impact on our most vulnerable citizens. In 1993, there were over 1 million cases of child 
abuse and neglect confirmed by child protective services in the United States. 
Approximately 1,300 of these children died as a result of their abuse. Additionally 
distressing is that over 57 percent of children under 12 who are murdered are killed by a 
parent (American Psychological Association, 1996). 



The issue of what constitutes child abuse or neglect remains undecided. 

According to a report by the National Center to Prevent Child Abuse (1997), “individual 
states continue to determine definitions of maltreatment...”. Most social service agencies 
generally recognize child maltreatment as physical abuse, physical neglect, emotional 
abuse, or sexual abuse (Maryland Department of Social Services, 1998); however these 
terms can be defined differently by child protective services workers in different states. 
What does seem clear is that many families who are reported for child maltreatment have 
a number of other presenting problems, such as, substance abuse, economic strain, and 
domestic violence (National Center to Prevent Child Abuse, 1997). 

The results of child maltreatment are evident by years of research. If the abuse is 
not deadly, it can lead to a number of physical and psychological problems. Repeat abuse 
can put some children at risk for physical problems such as brain and nerve dysfunction, 
developmental delays, and failure to thrive. Many studies have also shown that children 
who have been abused have a higher number of health problems, such as headaches, back 
pain, and gastrointestinal problems. As maltreated children grow to adolescence, it has 
also been shown that they are at a higher risk for low educational attainment, intervention 
by the criminal justice system, and suicide (American Psychological Association, 1996). 

This being said, there have been many identified strategies that assist in the 
prevention of child abuse and neglect. Some propose to build more nurturing families 
through parent education training and home visitation programs (American Psychological 
Association, 1996; Packard Foundation, 1999). A number of public and private agencies 
around the country are working to identify what works best. In fact, a recent report was 
published describing six national research studies evaluating the effectiveness of home 
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visiting as a strategy. Three of the programs included the prevention of child abuse and 
neglect as one of their goals, with two finding a significant impact of their intervention on 
reducing child maltreatment. Since each study targeted families with various 
characteristics, however, it is difficult to tell which interventions might be successful if 
used more broadly (Packard Foundation, 1999). 

Outcome Evaluation 

Outcome evaluation has become a focus for many non-profit agencies in 
determining how their programs work. With pressure from funding agencies, government 
legislators, and a competitive environment, human service organizations are looking for 
an opportunity to prove what they believe they inherently know - that they improve the 
human condition. In fact, a number of national level organizations, such as, the United 
Way of America and Goodwill Industries have developed manuals on how to implement 
outcome evaluation (Plantz, Greenway, & Hendricks, 1997; United Way of America, 
1996). 

Outcome measurement sheds a new perspective on the benefits of human service 
programs by providing an opportunity to answer the question, “What happens to the 
recipient as a result of the program or intervention” (Friedman, 1997; Plantz, et.al. 1997; 
Salzer, Nixon, Schut, Karver, & Bickman, 1997; United Way of America, 1996). While 
organizations were previously asked about how many people were served and how much 
service was offered, it was faithfully believed that the person receiving the service gained 
a benefit (Plantz, et. al.,1997). However with the advent of research results such as those 
experienced by Headstart, that put into question the short and long term benefits of a 
widely heralded program, this ideal is no longer taken for granted. 
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The approaches to outcome measurement established in the literature are varied, 
but there are some common elements. As discussed by Nancy McDaniel (1996), there are 
six key stages in outcome measurement: 

1) defining purpose and audience; 

2) identifying target outcomes; 

3) specifying measures; 

4) identifying sources of data; 

5) developing an implementation plan; and 

6) using the information to change the organizational culture. 

In addition, much of the discussion about the process of outcome measurement 
emphasizes the use of various stakeholders who can offer their unique perspective on the 
selection of outcomes (United Way of America, 1 996). A main benefit of outcome 
evaluation is to make programs accountable to a number of different audiences, thus 
improving an organization’s ability to communicate its results. 

Participatory Evaluation 

Along with the push to measure outcomes has come the emphasis to 
include stakeholders as participants in the evaluation process. In fact, the field of 
participatory evaluation (PE) has become such a hot topic that it was the sole focus of a 
1998 volume of the journal. New Directions for Evaluation . There are various forms of 
evaluation linked to PE, such as, stakeholder-based evaluation, transformative 
participatory evaluation, and collaborative action research, but all emphasize the 
importance of utilizing multiple perspectives from members of a group who have a 
decided stake in the program (Cousins & Whitmore, 1998; King, 1998). 
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The benefits of PE are many, and primarily focus on the usefulness of evaluation 
results. In their article on, “Ethical Dimensions of Stakeholder Participation and 
Evaluation Use”, Torres and Preskill (1999) discuss that stakeholder input can impact an 
evaluation in four ways: 

1) by designing the evaluation to be more responsive to stakeholders’ 
perspectives; 

2) by increasing the validity of the evaluation through the discussions that clarify 
the meaning of constructs and study findings; 

3) by increasing stakeholder ability to ask the right questions; and 

4) by educating stakeholders to be good consumers of evaluation. 

Although there continues to be some controversy over the objectivity and technical 
quality of PE, it has become widely used in the field and highly visible in the literature 
(Brisolara, 1998; Torres & Preskill, 1999). 

Agency History 

The Family Tree has been in existence since July 1997 and is the result of a 
merger between the local Maryland Parents Anonymous affiliate and the Child Abuse 
Prevention Center associated with the National Exchange Club to Prevent Child Abuse. 
Now the largest child abuse prevention agency in Maryland, The Family Tree was 
created to offer a broader continuum of services to families. The mission of the agency is 
to protect children by helping to develop healthy parents. The programs that support this 
mission include a 12 week parenting education course, an in-home visitation program, an 
adolescent parent program, and Parents Anonymous support groups. 
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The Positive Parenting Education Program (PPP) is a curriculum-based, 12 week 
parent education program conducted by trained and supervised community volunteers. 
Topics of instruction include positive parenting techniques, stress reduction, and self- 
awareness. A follow on service. Parenting And Teaching in the Home (PATH) is a staff 
and volunteer (8-24 visit) in-home support program provided to those who have 
participated in PPP and require additional assistance with improving family relationships 
and building parenting skills. Parents Anonymous is a program of on-going support 
groups facilitated by trained volunteer group leaders that assist parents in dealing with 
stress and improving parenting skills. The Parent Aides Nurturing and Discovering with 
Adolescents (PANDA) is a home and school based support and education program 
targeted at pregnant and parenting teens implemented by Parent Aides (trained mentors). 
These four programs make up the core of the Family Tree’s parent support services, 
however the agency also offers a 24 hour Stressline, as well as, community and 
professional training. 

Problem Statement 

Pressured by funding agencies and their Board of Directors to illustrate a positive 
impact of their programs. The Family Tree decided to implement a strategy of 
comprehensive outcome planning. Two of the four core programs, PPP and PANDA, had 
been administering the Adult Adolescent Parenting Inventory in order to evaluate their 
impact on parenting skills. Much of the data was never utilized however, because staff 
lacked the knowledge of how to analyze and interpret the information. As a result, the 
agency decided to hire an outside consultant to facilitate and support the implementation 
of an agency wide outcome-based evaluation plan. 
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Methodology 

The initial steps to this process included investigating the agency’s level of 
participation in evaluation to date, as well as, developing a strategy for outcome 
development. It was determined that most of the staff had no knowledge in the area of 
research and evaluation. In addition, the evaluation strategies that had been implemented 
were completed primarily to appease funding agencies through program reports. The 
information was not being used to help programs in their planning efforts. In fact, data 
analyzed for the PANDA program by an outside evaluator was of little use to the staff 
because they could not interpret the findings and there was no formal briefing about the 
results by the evaluator. 

Recommendations from this assessment supported the development of an agency- 
wide outcome strategy using a participatory approach. Because the agency had recently 
evolved as a merger of two separate organizations, the participatory process seemed 
particularly appropriate since it provided an opportunity for staff to come together to 
build consensus around an agency vision and purpose. It also allowed the opportunity for 
staff to become educated on program evaluation in order to become better consumers and 
users of outcome information. 

Utilizing the Results-based Accountability (RBA) model developed by Mark 
Friedman (1997), a planning group of administrative and program staff was convened to 
develop an outcome plan. Although the RBA model was originally proposed for use by 
communities who were developing state and local planning entities, the concepts 
established in RBA were also found to be useful in this endeavor. The main purpose of 
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the approach is to define results', or “conditions of well-being for children, families, or 
communities”(Freidman, 1996 p.3), and to quantify how well these results are achieved 
through measurable indicators. The process was interpreted for this agency as de finin g 
agency- wide results expected for families who were participants in services of The 
Family Tree. 

Beginning with a staff presentation on RBA and outcomes development in 
January 1998, The Family Tree initiated its first step in creating an overall agency 
outcome plan. The information on RBA was presented at a regularly scheduled staff 
meeting for all employees, so that even those who were not participating in the actual 
decision-making were informed about the work. Over the course of three months, a 
working committee of administrative staff selected by the Executive and Deputy 
Directors, met to brainstorm and prioritize agency outcomes. Strictly a democratic 
process, all ideas were voiced and noted on flip chart paper and priority outcomes were 
identified through multi- voting. 

Broad outcome statements were selected first, and identified as “results”. Because 
“results” are so difficult to reasonably define through indicators, a next level of sub- 
results were identified. These statements help describe more fully the intent of the results. 
Finally, a smaller group of staffbegan selecting indicators and measurement tools for 
each sub-result. Approximately two months was spent choosing indicators for each sub- 
result statement and researching and selecting appropriate measurement tools. 

The next step in the process was to link each of the four core service strategies to 
the identified results and principals. In July 1998, a meeting was held for all program 
staff presenting the newly developed outcome plan and training them on concepts related 

' The word result is used interchangeably with outcome. 



to program evaluation. Each program was then encouraged to identify its own outcome 
plan using the United Way of America’s logic model approach (1996), including agency 
results, sub-results, and indicators as appropriate. They were also encouraged to add any 
important program outcomes that were not necessarily stated in the agency-wide plan. 
Meetings facilitated by the outside consultant were held with each program individually 
over the course of approximately eight months until program plans were finalized. 



The participatory evaluation design used with this agency resulted in an overall 
agency outcome plan (see appendix 1), as well as, associated logic model evaluation 
plans for each of the Family Tree’s four core service strategies (see appendix 2). The first 
draft of the agency-wide plan included two main result statements, eight sub-result 
statements, and twenty-two indicators. After the program logic models were developed 
and linked to the agency- wide plan, the two main results remained, but only five sub- 
result statements and nine indicators were included. Essentially the program models were 
used to verify the original outcomes selected by the agency. 

Some of the original language used in the agency-wide plan was also changed. 

For example, the term “sub-result” was changed to “principal” to more accurately 
describe the statements as values. In addition, the term “indicator” in the agency level 
plan was changed to “program outcome”, since all of the individual programs had 
insightfully adopted the language from select sub-result statements as their client 
outcomes. Throughout the process, participants were encouraged to consider all of the 
outcome plans developed as dynamic and flexible with each new program year a test of 
their validity. 



Results 
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Lessons Learned 



Changing Mindset 

Throughout the course of this experience many things were learned. First and 
foremost is the fact that encouraging an organization to embrace evaluation means 
changing the mindset of “business as usual”. As people who work in human service 
agencies know, when faced with the overwhelming task of making a difference with little 
resources, the tendency is to move from task to task without taking the time to look at the 
big picture. The first barrier to overcome was to let staff know that their investment of 
time and effort was going to be worthwhile. In this case, the support from top 
administrators was a key to making this happen. 

In addition, when the outside consultant was first hired, many of The Family Tree 
staff felt that it was this person’s responsibility to evaluate their programs. Of course, 
with this notion came the fear that she was really the axe-wielding auditor who would 
most assuredly say each and every staff person should be fired and replaced. It has taken 
a great deal of time to break down this myth and to show the agency staff that they are 
the evaluators of their service. This individual empowerment was a direct result of using 
the participatory approach. 

Selection of Stakeholders 

Although encouraged to include stakeholders from various target groups such as 
the agency’s Board of Directors, consumers, and funding agencies, the administration of 
The Family Tree chose to only use staff as participants in the outcome setting process. 
With this came various pros and cons. The pros were: 

1) that staff were able to build some camaraderie around their vision for the agency. 
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2) that the outcome setting group was small enough to work efficiently, and 

3) that a strong rapport was built between the staff and the consultant. 

Cons to this approach included: 

1) the lack of outside perspective, and 

2) the low level of awareness about the process by the agency’s Board of Directors. 
It is expected that future steps for the agency will include educating the organizations’ 
many stakeholders on the outcome plan and encouraging feedback. 

Flexibility of Approach 

For this process it was important for the consultant to consider various approaches 
to outcome planning during the course of work with the agency. The initial use of the 
Results-based Accountability model was advantageous to determining the agency’s 
overall outcome model. It provided enough adaptability that it supported the development 
of the larger, overarching results, while supplying enough guidance to support the 
identification of indicators and measurement tools. When moving to the development of 
the individual program models however, it became important to communicate the theory 
behind the program as well as its outcomes and measurement tools. This supported the 
use of the logic model approach, and came coincidentally on the heels of a request from 
the United Way of Central Maryland for outcome plans of The Family Tree’s services. 
Overall, it was imperative for the consultant to stay informed of the trends happening in 
the field, as well as, recognizing the pressures on the agency from the outside. 

Support of Implementation 

Not surprisingly, it is in the implementation that the successfulness of all plans is 
revealed. With staff just becoming supportive of evaluation, came the difficulty of 
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making the outcome plan a reality. From ordering the measurement tools to organizing 
the gathered data, staff with little time available, were asked to take on additional 
responsibilities. It has been imperative that the consultant be available to support staff 
around how to make the implementation work. This experience supports the notion of 
“sustained interactivity” between evaluator and practitioner presented as a cornerstone of 
participatory evaluation (Cousins & Whitmore, 1999). This idea encourages the 
continuation of the relationship between evaluator and practitioner throughout the 
evaluation process. 

Feedback is Essential 

When the chips are down, it is the effort of making the information collected 
useful that creates the plan’s sustainability. Following the first quarter of data collection 
for one of the programs, the consultant analyzed the data collected and reported on the 
results at a program staff meeting. A dialogue on the interpretation of the results ensued 
and program staff were able to incorporate their new knowledge into their service 
delivery. One specific result of this process was the inclusion of a video to support the 
parenting skill of empathy in one of the program’s 12 training sessions. For the first time, 
staff were able to reap the benefits of their long commitment. 

Discussion 

Over the span of eighteen months, the culture of The Family Tree has changed 
enormously. In addition to hiring a new Executive Director, they have transformed from 
being a service delivery organization to a learning organization. An article written by 
Heather Weiss and William Morrill (1998) discusses learning organizations as those, 

“that have worked to apply a learning model” (p.4). This model includes: 
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1) identifying stakeholders to develop a plan, 

2) learning from experience and relevant research, 

3) engaging in monitoring and evaluation, 

4) learning from evaluation, and 

5) transferring lessons learned into program improvement. 

The initial intention of the agency’s directors in deciding to pursue outcome 
measurement was to meet the demands put on them from their funding agencies. What 
has resulted however, is the creation of an organization made up of staff who are no 
longer afraid to ask questions about their impact. The organization has also made a 
commitment to program evaluation by continuing to allocate more and more financial 
and staff resources to the effort. It is hoped that this is the beginning of a life long process 
of program evaluation and program improvement. 



O 

ERIC 



15 



13 



References 



American Psychological Association. (1996). Violence and the Family. American 
Psychological Association: Washington, D.C. 

Brisolora, S. (1999). The history of participatory evaluation and current debates in 
the field. New Directions for Evaluation. SQfwinterT 25-42. 

Cousins, B. & Whitmore, E. (1999). Framing participatory evaluation. New 
Directions for Evaluation. 80 (winter). 5-24. 

Friedman, M. (1997). A Guide to Selecting Results and Indicators: Implementing 
results-based budgeting. The Finance Project: Washington D.C. 

King, J. (1999). Making sense of participatory evaluation practice. New 
Directions for Evaluation SOfwinter). 57-68. 

Maryland State Department of Social Services. (1998). Factsheets on physical 
neglect, physical abuse, emotional abuse & sexual abuse. Maryland Department of Social 
Services: Baltimore, MD. 

McDaniel, N. (1996). Key stages and common themes in outcome measurement. 
The Evaluation Exchange, 2(3). 6-7. 

National Center to Prevent Child Abuse. (1997). Current trends in child abuse 
reporting and fatalities:The results of the 1997 annual fifty state survey. Prevent Child 
Abuse America: Chicago, IL. 

Packard Foundation. (1999). Home visiting: Recent program evaluations. The 
Future of Children. 9(1). 1-195. 





14 



Plantz, M., Greenway, M., & Hendricks, M. (1997). Outcome measurement: 
Showing results in the nonprofit sector. New Directions for Evaluation. 75(falll. 15-30. 

Salzer, M., Nixon, C., Schut, L., Karver, M., & Bickman, L. (1997). Validating 
quality indicators: Quality as relationship between structure, process, and outcome. 
Evaluation Review. 21(3). 292-309. 

The Family Tree. (1999). The Family Tree: Informational Report. The Family 
Tree: Baltimore, MD. 

Torres, R., & Preskill, H. (1999). Ethical dimensions of stakeholder participation 
and evaluation use. New Directions for Evaluation. 82rsummery 57-66. 

United Way of America. (1996). Measuring Program Outcomes: A practical 
approach. United Way of America. 

Weiss, H., & Morrill, W. (1998). Useful learning for public action. The 
Evaluation Exchange, 4(3/4). 2-14. 



er|c 



17 



15 



The Family Tree: Outcomes Matrix 1999-2000 



Appendix 1 









"pH 

C\J 



O 

S 

cu 

O 

u 



H 

§ 

Q 



X 

H 

< 

cu 

oT 

cu 

o. 

C /3 

CQ 

3C 

< 

Q 

< 

Pl. 



UP 

S 

o 

u 

H 

o 

s 

o 

o 

qC 

cu 



CQ 

u 

td 

t: 



cQ 
c/3 CQ 



C <1> 
C c/J 

•S JJ 

c/3 

c 

CQ O 
.&■ ®- 
‘c 3 o> 

r O 

CQ C 

a- O 

. u 

— CL 



< 

Ol 



O T 3 
^ C 

^ 'I 

UP c/3 

S'i 
£ S 

c/3 0> 

1^ 
u u 
T 3 

c/3 <!>' 

> 



3 o ^ 
•S- ° § 
.a E 
■C JJ 
<£S 



^ a> 
o> (M) 

■z? CQ 

•S c 

O CQ 

cL E 



< 

Ol 



T3 

C 

CQ 

C 

0> 

T3 



CQ 

^ 52 
*2 

S ® 

g. £ 

•n w 






cu a- 

* X 
(N « 



< 

Ol 

oT 

cu 

cu 



> c 

•p: cQ 

54^ 



ta 

u 

T 3 

C 

CQ 

CA 



fCQ 



crt P 

S I 

O c/3 
CQ 



CO 



c/3 



_ r 

CQ O 
O. CU 

:a §■ . 
t « -s 

CQ CQ c 

Di <u 0 > 
. CO *r; 
3 



O 

c= 

— 

'o 

‘> 



o 

£ 

o 



< 

Cl 

U 

5 

Cl 



CO o 

i 

1 - 

I S 

^ T 3 
^ cQ 



t> 

it 



CQ 

u 



3 

£ 

£ 

o 

o 



£ 

CQ 

UU 

QQ 






o 

o 

0> 

4= 

0> 

> 

CQ 



0 > 0 > 
CQ O 

^ e 

u S 



£ 

CQ 

U. 

&£) 



H 

:p 

c /3 

S 

>- 

u 

§ 

o 

< 



3 

t: 

3 

T 3 

C 

CQ 

b£l 



o 

cv 




PANDA OUTCOME DIAGRAM 



Appendix 2: 



ERIC 



cv 



c 



ti 

cd 

O 

-n 



CO) 

c 



Cd 

^ > 
E "o 
S cj 

C 

fP cd 

w jp 

O. 0> 

E 

O o 

^ J= 



c ^ 

^ & 
h 3 
cd o 

Cd o 
O. 

E ^ 

TO 3 
C M 
QO ^ 

(D 



o o 
^ E 



Cd 

h 

u 

u 

73 

3 



0> 

ti 

Cd 

O 

j= 



o 



o. 

o 

-n 



o 



E 

CL 

E 



CL 

’T3 

C 

<L> 

Cd 

o 

j= 



o 

ti 

Cd 

O 

-n 



(aO 

c 



2^ CL 

2 ^ 

CL o 



o 

E 



td 

.B- 

’3 

'€ 

Cd 

CL 

O 

-n 

c/} 



•o 

(O 

cd 

JO 

*o 

o 

-n 

o 



CL 



t: 

Cd 

CL 

O 

-n 



-o £4) 
3 t 


LiM 

o 

JZ 


CO 


LiM 

O 


CO 


^ O 


cd 


CL 


TO 


CL 


3 CL 


C3 


3 


n2 


3 


Cd CL 
3 3 


00 


e 


00 


e 


00 CO 


3 


00 


3 


00 


Urn >\ 


O 


r. 


O 


t: 


CL2 




o 




o 


C4- « 


C4^ 


CL 




CL 


O V 


o 


CL 


o 


CL 


% > 




3 

CO 




3 

CO 



o c= 
CJ o> 
^ 4> 

o ^ 

Qu E 



Cd 



< 0) 

^ t_ 



x> 

— Q> 

U a> 
o> ^ 

|2 

^ C/3 

3 



O 0> 
O 0> 

o' c 



3 C 



O. 

3 

e 

&0 

I- r 
d> o 

« CL 
_ O. 
3 



c/3 

O 

-o 

c 

a> 

a> 

a> 

o> 

> 



CL 

3 

e 

&0 

t: 

o 

CL 

CL 



o -c 



CL^ 

*o 5 
0) 

»-3 

•? ij ^ - 

E S S E 

0 

1 



^ ^ fc 3 

fc- Cd 



Cd 

o 

3 
*U 
0> 

c 
u 

cd 

CL 

— c 



ii o 
c 

- Cd 

J* -o 

Cd « 



QD 



C 

u 

cd 

CL 

*o 



cd 

g- 5) 

^ 2i os 

t« Q, ^ 

0 g - 

5 

1 &i2 

|c2 g 

0^0 



u 

0> 

u. 

o 

E 

75 

CQ 



go’s 

.E 



o S 

Cd T3 

-i 

C .3 

Cd T3 

rt S 

c Cd 
W), 

0) ' 



ON 

5. T 



eN 



o B 
■§'g 

S3 

-o 



3 ® 

c o 

Cd Vi 



Cd 



Cd 

CL^ 



O 

o 

^ ON 
. ON 

D ^ 



CL « S 

O 3 T3 

(D od 



o 

c« ^ 

-9 iS 
^ 0> 
o ^ 

o ^ 
-c 
u 

y) Cd 



o 

CL . . 

9- C ^ 

o 

^ SP 

o, i2 

o O C 

^ CL O 

^ O w 

j—* 'trJ c/i 

o ^ O 
W)^ o 
o JC 

t2 'S o 

^ cd C/3 



ll 

c ^ 

■B c 

cd O 
O CL 
3 CL 
-O 3 

fl> Cfl 

5 

2 S 2 

00 ™ u 

£ C « 

^ o,^ 

Vi CL ^ 
k. o 

<i> E 
-^ 2 

^ -S O 

bo c o 

C O -C 

3 C CJ 

g E W 

>" Cd ^ 



<n 

Cn| 

I 

NO 



O 

S' 



u 

Urn 

Cd 

CL 

JJ 

Cd 

E 

3 

O 



CL 

O 

Oh 

t> 

£? 

Cd 

H 



o 

00 

Cd 

3 

Cd 



3 s 



E g 

Cd 3 

Ofi V 

o = 

£ cH 



d> 

OD 

cd 



E 

Cd 

e ^ 

ou E 



< «3 



o 

E 



■ c 

Cd 

CL 



o 

E 



(D 

CL 



O 

00 



U 

o 

u. 

O 

E 

GO 



o 

o 



0> 

CL 



<Ti 



3 

O 

U 

a 

CL 



0> 

0> 



si- c ^ 3 

= ^ ^ 

3 O G> O 



3 ^ 

0> 



E 

Cd 

k. 

00 

p 



Cd 

Ll 

00 

3 

3 

O 

>- 



o 

> 



o 

E 



CN 

00 



3 

O 

U 

b 

o 

c/3 

> 

-3 

< 

u 



Cd 

GO 



O 

E 



CN 



CJ 

3 

3 

O 

U 



o 

E 

CL 

> 

0> 

Q 



3 

O 

U 

d 

0. 



cd 

> 



E 



00 

3 

3 

O 



< 

Q 

< 

a. 

I 

u 

2i 

E ^ 

I §> 

3 o 

GQ ^ 

Gl 

00 

C Vi 

.3 *- 

£ £ 



O 

O 

oo*' 



3 

O 

u 

a 

a. 

00 



3 

Ll 



Cd 

GG 

o 

• • M 

■g fe 
1 = 
Q . 

s 

S.U 

C/D o> 

00 o 

• s E 

o> *s3 

2 GQ 



T3 

0 
c/3 
Cd 

X) 

1 

o> 

E 

o 

a: 

CO 

•E 

§ 

•o 

d 

o o 

U £ 



§ U 

U ^ 

r-3 

CL w 



CM 

CM 



* for entire PANDA program 



in 

OJ 



UJ 

s 

b i-J 
U. U 

C/3 H* 




H 

2 

UQ 

S< 

;2 h 

sS 






c 

0> 






c 

0> 



c 

0> 



CO ■<-» 



CO 

> > 

.2 2 

:l ^ 
. > 
CO 0 ^ 



CO 

> > 

1 - 

:s ^ 
. > 
CO o> 



o 

CO 

V 

CO 

ed 



o 

CO 

0> 

CO 

cd 



M !S ■« !S 
> > > > 
•3 -3 

I b| b 

.= O) 0> 

. > . > 

CO 0> CO 0> 






CO .ii 

> > 



.ti 'y 



.= o> 
. > 
CO 0 ^ 



CO 

cd 



> 

> 2 

1 &' 
.ti o> 
c > 

•— o 

cd ^ 



’m .22 

> > 

•s &' 

.= o> 

. > 

CO 0 ^ 



o 

CO 

0> 

CO 

cd 



0) 

CO 

cd 



•O 

c 

CO cfl 

’ 2 o ’.5 

lo^i 

• E 

.= u 
. > 
CO 0 ^ 



u 

CO 

u 

CO 



H 

UQ 

S 

§ 

c/3 

< 

UQ 

S 



0> 














o> 






0> 


o> 














0> 














0> 






0> 


0> 










To 




X 




u 




u 






X 


o 




X 


*3 


o 




o 




o 


o 


CO 




2 




2 






CO 


CO 




CO 


CO 


CO 




CO 




c/3 


CO 

'W 


"E 

o 




3 

O 




3 

O 






*3 

o 


3 

O 




To 

o 


To 

o 


3 

o 




3 

O 




SP 


3 

O 


00 




o 




u 






S) 


o 




S) 




o 




o 




u 


u 






u 

o 




u 

o 








u 

o 








u 

o 




u 

o 




X 

3 


u 

o 


E 


CU 

< 


3 

0> 




3 

0> . 


CU 

< 


CU 

< 


E 


3 

0> 




E 


E 


3 

0> 




3 

U 




G> 

CO 

o 


'W 

3 

0) 


<42 


< 


E 


E 

u 


E ^ 


< 


< 


c2 


E 




2 


a 


E 


E 


E 


E 




E 


3 


cd 


cd 


cS 


E cS 


cd 


cd 


cd 


cd 


c2 


cd 


cd 


cd 


c2 


cd 


c2 


zi 


X5 



c/3 

OC 

o 

H 

< 

U 

5 

2 



0> 

.2 

'C 

o. 

s 

o. 

o. 

CO 

a 

00 

c 

’S 

CO 

o> 

E 



§ ^ 
.2 2 
CO 

•2- b 

.2 

ti ^ 
E « 
a- To 

c « 

0> CO 
*0 CO 

O a> 

-J? 

«2 IE 

c 

S-;S 

!2 ^ 
t: CO 
Cd 2> 

o. vn 
o 1 

Sp o 

^ CO 



CO 

c 

0> 

E 

o. 

"E 

> 

o> 



c .ti 

O CO 
<1> ’> 

XJ ^ 

2 b 

o 5> 

S) ^ 

^ < 
Q> 0> 

o 

U 

CO o 

2 .2 
cO 
O 



^ X) 



CO CO 
O. CO 
.2 § 
i 3 

o D, 

X 
0> 



c 

0> 



o 

u 

’5 

X 

00 

.E 

o 

"ob 

0> 

c 

u 

O 

00 



Xi 

CO 

'S 

c 

t: 

o 

o. 

Qi 

o 






.B* 

’3 

i;l 

a. > 



Sp CO 
^ 0) 



0> 

E 

o 



o 

CO 

0> 



c 

o 



o 



00 

,c 

*c 

CO 

0 

oT 

> 

’co 

3 

X 

cO 

1 

C 

o 

c 

T> 

•o 

o 

E 



§ 

Q, 

’o 

'€ 

CO 

o. 



o 

p2 

To 

o 

CO 

X 

3 



CO 

o. 

E 

0> 

0> 



c 

o 

0> 

> 

o 

X 

CO 



0> 

00 

2 

0> 

> 

CO 

0> 

o 

o 

CO 



CO 

_ 

t N 

CO ” 

ir 

o u 



o. 
o 



CO ^ 

“u < 
: < 
0> 



> 

(N 

b 

> 

0> 

Ou 

< 

< 

a> 



c 

o 



o 

u 



0) 

> 

s 

o. 

E 

o 

X 

CO 

C 

CO 

o, 

'o 

r 

CO 

CL 

o 















O 








u 








CO 




(N 












>» 




^3 


E 






!S 


CO 


.S 








2 


CO 




p 








o. 


■| 


'E 


o 




CO 


> 


00 


3 

U 


'E 


(N 


3 

0> 


E 

>» 


o 

00 


.E 


E 

>» 


"q. 


3 

u 


2 




E 


E 


’5 


o. 


u 




CO 


E 


u 




*E 


d> 


O 


o. 


o 




3 


E 


00 


o 


.2 


u 


CO 


E 

u 


E 

o 




*0 

U 

U 


3 


3 




3 


T3 

O 


•o 


CO 

o 

3 


g 

CO 


<^ 


E 


ns 

G> 


CO 

X 




•o 




1 . 


3 


o> 


U 


o 


0> 




0> 


o 


£ 


o 

H 


E 


E 


o 


3 

u 


o 


o 




X 


X 



Q, 

’o 



a. 

^ *> 



o 

0> 

CO 

o 

o 



i2 



O) 

o 

c 

u 

3 



CO 



o 

h 

c 

o 

o 

X 

IS 

o 

0> 

CO 

3 

r 

o 

o. 

u 



a, 

’o 

i 

o. 

sp C 

^ Qi 



U 

c 

CO 

I) 



3 

o 

&< 

0> 



o 

> 

cO 



o 

3 

o 

•a 

o 



E 

u 



£? 

u 

X 

3 

0) 

CO 

O 



3 

O 

QJ 

Urn 

o 

o 



> 

s 

o. 

E 

o 

X 



CO 



CO 



CO 

3 ^ 
CO <N 
Q, 



3 

E 

>% 



u 

3 

I 



00 

E 

’> 

S 

o. 

E 



•o 

£ 

3 

O 



§ i2 



o 


G> 


G> 


o 




.2 •> 


*€ 

CO 

O. ' 


CO 

O 

.3 


CO 

O 

3 


€ 

CO 

o. 


0> 

> 

o 


is 












<*- b 


o 


O 


O 


o 


*E 


o fe 


vP 




NP 


Np 


Q 


sP > 










CO 


^ 0^ 



Q 

® t3| 

w w n 

2 o H 
O ^ D 

a: a- o 



§ 

o. 



o .2 

00 3 

3 p 



no 

3 

3 



^2 3 S « 



cO CO 
O. U 
3 
CT 



rcd£ 2 .>-- 

™ Gi Gi .ti S 

Cu > -o CO 



3 

3 



^ u 
42 > (N 
3 « — 

5 -w CO 

.S- o 2 .ti 

o X S .S£ 

^ Z > 

CO Q> CO U 

^ o -o g 

« 3 CO O 



te J= 

42 <S ^ ^ 
g -o 42 o c 

o. S § & S 



•- « E 

.E ^ M Q. 
t- ^ o c o 
c 3 ^ O Ti 
a. § e S2 > 

. 3 0> 0> 

(N ^ Q. O.T3 



T3 « 

^ g CO 

<u g 2 
'g S, S 
-2 ^ S 



i 

CU 



CO ^ 



0> _ 
O 3 CO 

E o 5 s= 

”0 D- CO 




cv 



!> 

CVJ 



Q 

Z w 

< 5 

U, 

U, UQ 

< 5 

C/3 H 



-g 

* 

rz: o> 

t ^ 

5 CO 

c/3 -O 



4> 

^ O 

« E 
S 3 

CO 

s < 

.s < 



CO 

r- 

o a> 

CO ^ 

"eo c 

S:s 

0> ^ 
^ c 
O 

g S2 

:i S. 



H 

§ 



U3 

o: 

D 

c/3 

< 

5 5 




dI «2 >S 



Z, O <0 

^ ^ *13 •4-* 



c 

•C O 
Q. 



- £ 

*3 S-® 

Q> eo CO 



c Q.-= 

w ^ ^ ^ ^ c 

^ ^ U > O ^ 

C W3 S ^ 

^ ^ - 



I § i « 5 
J “!. 3 M jj <N 

%t.^-c = .B 

" - '5 S 



c 



J 

•o 

i 



c 

O <N 

CO 

QJ jC' 
CO 



- U — 3 

Z C/3 * T3 



E 

§3 

O 

D. 



i J 



5b g 



*E W V3 

5 CO fc, s>» 

|3 &| 

c£ c2 vS ^ 



H 

§ 

s 

§ 

D 

< O 

lg 



cu 

< 

< 



cu 

< 

< 



cu 

< 

< 



— 

*« 

o 

c/3 

bO 

u. 

<H 

X> 

c 

a> 

CO 

O 

0^ 



C/3 

0^ 

< 

u 

5 

z 



NO 

©N 



c 




o 








« 








o 




0> 




Q. 


CO 


X 

0> 


(L> 




JZ 


To 


o 




3 


c 


0> 


0> 




E 


3 


a, 

jO 


CU 


a> 


< 


> 

o> 


< 


•o 


0> 



c 

Oi t- O 

•B ^ 'S 

e ® 2 

o — 

« S !e 

o ^ E 

*§ S "rt 



O « 

<L> JZ 
OJD -*-* 

^ O e 

o> 5 

> .2 

'rt *'5 

0> U S 

U CO ^ 
O X3 .22 
CO CO .t; 

O >% E 

HE 

^ « 

i« & M 
C C « 

s « 

.9- « -5 

u •£ CO 

’€ c " 

S.S “ 

> a. 
o O C 
-9 ^ 

^ cO ^ 



O 

u 



a> 



CO 

Q> 

U ^ 

> ^ 

2 S 

Q. Q. 

•- 3 

0 U 

*1 

1 

t| 

f < 

Q. <u 
J= 
O ^ 

nO C 

^ O 



o 

CO 

sa 

-C o 

o g 

& ia 
.iu 

o i: 

JZ U 
^ CO 

•M Up 
c o 
CO ^ 
O. C 

o s 

CO 



O. 






.3 


o 




vO 


C 




O 


*0 


CO 

CO 


c 


0> 


3 


C 


U 

CO 


0> 

A 


CO 




c 




o 




E 


CO 


u 

*0 


0) 


CO 




C 


CO 


3 


O 


.2- 


O. 


‘o 


U 


‘2 


o 


E 

cu 


E 

o 




Urn 


(N 


Q. 



H 

U 

U3 

I 

cu 

Q 

U 

c/3 

< C/3 

CQ u 

J s 
O o 
O u 
X t: 

^ S 

c/3 O 



“ ^ 

o> .C 

> •- 
CO CO CO 
JC O V 
. CU 3 
: cu- o- 

i o *£ 
i2 «yo-c 
S -S g 

« -o ^ 

. 2 - c ojD 

o « c 
111 
°:l i 

— 3 Q. 




CD 

CM 



participants are defined as those who have attended at least 4 meetings within a 2 month period 
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